Stochastic Gradient Descent with Polyak’s Learning Rate

Stochastic gradient descent (SGD) for strongly convex functions converges at the rate $$\mathcal {O}(1/k)$$ . However, achieving good results in practice requires tuning parameters (for example learning rate) of algorithm. In this paper we propose a generalization Polyak step size, used subgradient methods, to stochastic descent. We prove non-asymptotic convergence with constant which can be better than corresponding optimally scheduled SGD. demonstrate that method is effective practice, and on optimization problems training deep neural networks, compare theoretical rate.